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Abstract 

Background. The critical review is an important, though not 
always well understood and correctly applied element of Life 
Cycle Assessment studies. It is the intention of this paper to 
analyse the relevant standards and to present personal experi¬ 
ences in conducting critical reviews. 

Results and Discussion. A peer review for LCA-studies was first 
proposed in the SETAC guidelines 'A Code of Practice' (1993). 
The ISO standard 14040 (1997) describes three types of 'Criti¬ 
cal review' which are optional in general, but mandatory 'for 
LCA studies used to make a comparative assertion that is dis¬ 
closed to the public'. For this purpose, the most comprehensive 
form according to section 7.3.3, the panel method, has to be 
('shall' in ISO terminology) be used. From personal experience, 
this method is found to be the most frequently performed in 
practice (60%), the average panel size being three experts. Large 
panels with more than 4 experts are rare. 

Recommendation. Personal experience leads to supporting the 
recommendation by SETAC of the accompanying or 'interac¬ 
tive' critical review, which should be preferred, over the review 
'a posteriori', which offers considerable risks in regards to the 
duration and costs of an LCA study. ISO 14040, on the other 
hand, does not recommend one or the other way of performing 
the critical review. 


Keywords: Critical review; comparative assertions; ISO 14040; 
SETAC code of practice 


Introduction 

Peer review in LCA is completely different from peer review 
in scientific journals [1,2]. It is not anonymous and - quite 
to the contrary - requires a close and open co-operation 
between the commissioner of an LCA study, the practitioner 
(very often a consulting firm) and the reviewer or the review 
team (panel). It is also much more time consuming and can 
be compared to review procedures of research projects, e.g. 
by governmental research organizations. Furthermore, also in 
contrast to journal review, not all data and information scru¬ 
tinized will be published. The reviewed study is often not pub¬ 
lished at all and may serve political or marketing goals rather 
than scientific or technical ones. These political and commer¬ 
cial uses of LCA are foreseen in ISO 14040 [3], albeit not as 
part of the LCA, but rather as possible applications. 


* Presented at the Discussion Forum 23 'Quality Issues in LCA Software, 
Projects and Peer Review 1 . September 23, 2004, ETH Zurich (see p. 158). 


1 Peer Review according to SETAC Code of Practice 

The first recommendation of a peer review in LCA was given 
in the SETAC guidelines for Life Cycle Assessment (1993) 
with the sub-title 'A code of practice' [4]. One of the rea¬ 
sons for the emphasis on peer review was a considerable 
misuse of comparative LCAs in the early times of LCA (ca. 
1970-90), e.g. by using deliberately incompatible system 
boundaries and deriving evidently false claims from LCA- 
based results. This misuse also seems to have been the rea¬ 
son for other restrictive recommendations in the SETAC 
document [4] and, later, in the standards ISO 14040-43 

[3.6.10.11] . Examples for such restrictive advices are the 
interdiction of weighting (valuation) in comparative asser¬ 
tions disclosed to the public (ISO 14042 [10]) and the low 
ranking of subjective or arbitrary allocation factors (instead, 
the clumsy system enlargement is ranked on place a/2 in 
ISO 14041, 6.5.3 [6]) 

The 'Code of Practice' gives two main reasons for performing 
a peer review: to enhance the quality and to enhance the cred¬ 
ibility of an LCA study. Stricter than the later ISO standard, 
SETAC recommends an interactive or accompanying review. 

Finally, SETAC gives advice on how to perform the review, 
whereas ISO deals mostly with the formal aspects. As with 
the rest of the Code, it can be stated that a very important 
input to the ISO 14040-43 series of standards was given. 

2 Critical Review according to ISO 14040-43 

The international standardisation process started in the same 
year in which the SETAC Code of Practice was published, 
i.e. in 1993 [5], and the first result was the framework norm 
ISO 14040 [3], published four years later. For the 'critical 
review', as it has been called since, ISO 14040 is the decisive 
standard; in the other LCA-standards ISO 14041-43 

[6.10.11] , the critical review is only mentioned occasionally 
(see below). 

The critical review is first mentioned in paragraphs 5.1.2.4 
and 2.1.2.5 (critical review considerations) [3]. In the last 
mentioned section one reads: 

"Critical review is a technique to verify whether an LCA 
study has met the requirements of this international stand¬ 
ard for methodology, data and reporting. Whether and how 
to conduct a critical review, as well as who conducts the 
review, shall be defined in the scope of the study." 1 

1 Here and later in the text: bold letters in citations are inserted by the 
author in order to emphasize important issues. 
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This wording defines the main purpose of a critical review 
and establishes the formal link with the scope definition 
which is elaborated in more detail in the subsequent stand¬ 
ard 14041 [6]. The standard 14040 says further: 

"In general, critical reviews of an LCA are optional and may 
utilize any of the review options outlined in 7.3. A critical 
review shall be conducted for LCA studies used to make a 
comparative assertion that is disclosed to the public and shall 
employ the critical review process outlined in 7.3.3." 

The first sentence reads harmless: critical reviews are 'op¬ 
tional' and there are several possibilities to conduct a re¬ 
view. The second sentence, however, strictly demands for a 
critical review in its toughest form, in a wording which be¬ 
came famous in the LCA community. 'Shall' means in the 
ISO language 'must without exception' and indicates a very 
strong wording. This message is repeated in Section 7.1 in a 
chapter which is entirely devoted to critical review: 

" ...In order to decrease the likelihood of misunderstandings 
or negative effects on external interested parties, critical re¬ 
views shall be conducted on LCA studies where the results 
are used to support comparative assertions." 

The emphasis of this paragraph is on avoiding 'misunder¬ 
standings', etc., an understatement. Much to the relief of all 
those conducting critical reviews according to section 7.3.3, 
we further read that " the fact that a critical review has been 
conducted should in no way imply an endorsement of any 
comparative assertion that is based on an LCA study". Again, 
this wording shows the firm determination of the authors of 
ISO 14040 to avoid any misuse of LCA, e.g. in marketing or 
for policy making, by the use of LCA-based claims that are 
not well and scientifically founded. 

The most important section 7.3.3 is formulated in an in¬ 
triguing way, insofar as there is an apparent contradiction 
between the title and the text of the section. The title of 
paragraph 7.3.3 is 'Review by interested parties'. The stand¬ 
ard clearly says that the commissioner has to 'select' (i.e. 
invite) an 'external independent expert' to act as chairper¬ 
son of a review panel: 

"An external independent expert is selected by the original 
study commissioner to act as chairperson of a review panel. 
Based on the goal, scope and budget available for the review, 
the chairperson selects other independent qualified reviewers." 

According to the following sentence, however, the inclusion 
of 'other interested parties' besides the experts is optional, 
as can be seen from the very weak 'may include': 

"This panel may include other interested parties that will 
be affected by conclusions drawn from the LCA study, such 
as government agencies, non-governmental groups, or com¬ 
petitors. " 

As will be seen in section 5, large panels are rare in practice 
for financial reasons and probably also for the sake of con¬ 
fidentiality. Comprehensive LCAs used in the context of 
political decisions often have advisory boards including in¬ 
terested parties, such as representatives of industry associa¬ 
tions, environmental and consumer protection groups, etc.; 


therefore, there is no or less need to include them in the 
review panel as well (e.g. [7,8]). The critical review panel or 
at least the chairperson is frequently invited to attend the 
advisory board meetings. In another variant, project pres¬ 
entations by the practitioner are combined with advisory 
board meetings and followed by a review panel meeting in a 
small circle. Fava and Pomper [22] describe a tiered, inter¬ 
active critical review and very positive experiences gained 
in this procedure. 

The question, whether an 'a posteriori' or an accompanying 
('interactive') [4] critical review is to be preferred, is not 
addressed in ISO 14040. Compared to the SETAC guide¬ 
lines [4], ISO 14040 is less demanding and more flexible in 
this point. 

There is a strong statement in the standard that the review 
statement, the review report and any written comments by 
the commissioner and/or the practitioner "shall be included 
in the LCA study report." 

There are two other, less demanding modes of critical re¬ 
view: The internal review (7.3.1) and the external review by 
an independent expert (7.3.2). The review following 7.3.1 
could be conducted via internal quality control in the com¬ 
missioning company. It is said in ISO 14040 that the re¬ 
viewers have to be independent on the LCA-team (be it ex¬ 
ternal or internal). For the review following 7.3.2, in principle 
the same requirements apply as for chair persons in the panel 
mode (7.3.3), although the activity is in general less visible. 
The review statement is a part of the LCA study report in all 
variants of performance. A review according to 7.3.1 or 7.3.2 
may be published together with the LCA study report, if the 
study does not contain comparative assertions (requiring a 
review according to 7.3.3), e.g. in some cradle-to-factory 
gate studies or LCIs (e.g. [9]). 

The main duty of the reviewers is laid down in ISO 14040, 
'General description of critical reviews' (7.1) and applies to 
all forms of critical review (7.3.1 to 7.3.3): 

"The critical review process shall ensure that: 

- the methods used to carry out the LCA are consistent with this 
international Standard; 

- the methods used to carry out the LCA are scientifically and 
technically valid; 

- the data used are appropriate and reasonable in relation to the 
goal of the study; 

- the interpretations reflect the limitations identified and the goal of 
the study; 

- the study report is transparent and consistent." 

These five points clearly define what has to be studied in the 
critical review and also serve very well in structuring the 
core element of any review report according to ISO 14040, 
the review statement. 

The critical review is also treated in a little section or para¬ 
graph in each of the detailed standards ISO 14041 [6], 14042 
[10] and 14043 [11]: 

- ISO 14041: 5.3.7 

- ISO 14042: 10.3 

- ISO 14043: 9.2 
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In most cases, the text only reaffirms the prescriptions of 
the framework standard 14040 [3], whose relevant sections 
are quoted. Only the Life Cycle Impact Assessment (LCIA) 
standard ISO 14042 [10] gives in section 10.3 a new state¬ 
ment on the necessary qualifications of the reviewers related 
to environmental sciences: 

"For LCIA, the expertise of reviewers in the scientific disci¬ 
plines relevant to the important categories of the study, in 
addition to other expertise of interest, shall be considered." 

This item is important, since it is the only clear statement 
that the critical review requires expert knowledge in the en¬ 
vironmental sciences and is by no means a routine work. In 
this connection it should be mentioned that a 'certification' 
of LCA-studies, as occasionally suggested and even prac¬ 
tised (e.g. [12]), has no foundation at all in ISO 14040-43. 

3 Effect of Critical Review on Quality and Credibility 

The following section mostly deals with critical reviews ac¬ 
cording to ISO 14040, section 7.3.3, but similar positive 
effects can also be expected from weaker forms of review. 

Many prescriptions of the standards, especially in ISO 14043 
(interpretation) [11], aim at securing the quality of LCA- 
studies. Since the critical review has to scrutinize the appli¬ 
cation of the standards, it automatically enhances the qual¬ 
ity of the LCA. This can hardly be seen in the final review 
reports, since most suggestions made by the panel members 
are taken into account by the practitioners concurrently. In 
nearly each review, the panel members produce long lists of 
errors and suggested improvements on the basis of their per¬ 
sonal experiences and expertises. 

Frequently, the review team members agree in a sort of spe¬ 
cialisation, e.g. 

- Compliance with standard/goal and scope; 

- Data/inventory (LCI); 

- Life cycle impact assessment (LCIA); 

- Management of the review process. 

Each member of the panel reads the whole report, but the 
depth of the critical reading may and should differ. Some¬ 
times, one member of the review team is a data specialist for 
the product or production process which is in the focus of 
the study; this member has not necessarily to be an LCA 
expert, although it is usually better that all members are 
familiar with this technique. 

Since the quality to be achieved in an LCA study depends on 
the goal and scope definition, there is no general scheme for 
assessing the quality. This has to be elaborated in discus¬ 
sions within the panel and/or with the practitioner and pos¬ 
sibly the advisory group and the commissioner. 

The quality of the review as well as its credibility depends to 
a large extent on the communication and co-operation be¬ 
tween the three players: commissioner, practitioner and re¬ 
view panel (the 'review triangle') . 

Especially the chairperson of the review panel, the repre¬ 
sentative of the commissioner, and the LCA-project leader 


have to co-operate closely. From my personal experience, I 
can say that it is more difficult to conduct a critical review if 
the commissioner does not participate in the review process 
actively (the practitioner always has to, only in case of an 'a 
posteriori review', the practitioner may not be available). If 
the 'review triangle'works, and especially so in an accompa¬ 
nying review, the commissioner hardly has a chance to push 
the study in a direction which might please his marketing peo¬ 
ple (hired gun effect). The review team effectively backs the 
practitioner and thus contributes to the credibility of the LCA- 
study. On the other hand, evidently, it also supports the com¬ 
missioner in his wish for unbiased high-quality work. 

Not only the practitioners, also the reviewers can loose their 
credibility and therefore both are highly interested in a correct 
performance of the LCA and the critical review process. Fi¬ 
nally, the commissioner gets more for his money even if the 
results may differ from anticipated results, since only well per¬ 
formed LCAs can result in long-lasting improvements. 

One important aspect concerning quality and credibility is 
access to original data. Actually, the confidentiality of sensi¬ 
tive data (see a recent criticism by Frischknecht [13]) is one 
important reason for conducting a critical review. Since not 
all data are published, or only in a highly aggregated form, 
the reviewers have to judge their quality and appropriate¬ 
ness by random samples. In order to do so, they have to 
have access to the data: commissioner and practitioner must 
provide this access. This may be complicated if third parties 
provide data and are not prepared to reveal confidential is¬ 
sues. This problem was encountered in an early review 
[14,15], where a subcontractor did not allow full access to 
some crucial data. This resulted in a delay of several months, 
additional sensitivity analyses and other activities not fore¬ 
seen in the work plan. A lesson to be learned: sub-contractors 
have to be included in data transparency for the review team 
or at least for one member of the panel responsible for data 
quality. In my anecdotic case, the difficulties were finally over¬ 
come, mainly since the 'review triangle 1 worked perfectly. 

In another critical review for an LCA-study [16,17], during 
which difficulties arose, the commissioner did not commu¬ 
nicate with the review team, at least not directly. The trian¬ 
gle, in that case, collapsed into a line opposing rather than 
joining practitioner and review panel. However, also in this 
case the critical review process was finished correctly and, 
as an additional benefit, the end-of-life LCI-models devel¬ 
oped by the practitioner were fully published on request by 
the reviewers [18,19]. 

4 Effect of Critical Review on Project Duration 
(including cost) 

Before discussing the effects of the critical review process on 
duration and cost of LCA-studies, one has to remember that 
there are two kinds of critical review: the accompanying or 
interactive one and the review a posteriori. The decision what 
kind of critical review will be performed may have conse¬ 
quences for the duration and finally the cost of the LCA-study. 


100 


Int J LCA 10 (2) 2005 



Commentaries 


The Critical Review Process 


In an accompanying critical review, the review team (in case 
of 7.3.3) is - to a certain degree - part of the project team, 
although a privileged one due to its independent status. The 
reviewers can and often do influence the study, except the goal 
itself: this is explicitely excluded in ISO 14040 section 7.1 [3]: 

" Since this International Standard does not specify require¬ 
ments on the goals or uses of LCA, a critical review can 
neither verify nor validate the goals that are chosen for an 
LCA, or the uses to which LCA results are put." 

The sooner the review team is included, the better for the 
duration of the project. The best time is when the draft goal 
and scope chapter is finished. In this phase it is still possible 
to propose major changes in the methodology, e.g. the allo¬ 
cation rules or strategies to avoid allocation. Since the re¬ 
view team can influence the study (provided the suggestions 
are accepted by practitioner and commissioner) the results 
of the study will in general be acceptable and only minor 
changes will be required in the stage when the final report is 
written. In this case it typically takes only a few weeks (real 
time) to complete the review report, if the remaining im¬ 
provements are quickly performed by the practitioner. The 
final conclusions, recommendations, the executive summary 
(if there is any) may be debated fiercely within the 'review 
triangle' in the final phase and take additional time. 

In the case of a critical review a posteriori the situation is 
completely different. The review panel is in most cases con¬ 
fronted with the draft of the final report, where it is too late 
to change anything fundamentally. The first comments by 
the panel may be very critical. Now, in the final phase of the 
project, the project budget is in most cases spent, practi¬ 
tioner and commissioner have to negotiate who pays the 
additional work that might be required (personal note: my 
memory as project leader tells me that it is often the practi¬ 
tioner who pays). The review team is not a member of the 
overall project team and therefore can criticize the whole 
project and any amount of details 'from outside' without 
any psychological restraints. This is, of course, also the strong 
side of the review a posteriori : the fresh look from outside 
and the lack of restraint in being critical. 

The delay in this type of critical review may amount to sev¬ 
eral months or even longer. It is a great risk for both the 
practitioner and the commissioner, which cannot be calcu¬ 
lated at the beginning of the project. The accompanying criti¬ 
cal review may involve a few reviewer days more, but it can 
be seen as a kind of 'insurance' against a possibly disastrous 
final phase. 

This brings me to the question of the expenses needed for a 
critical review. I can try to do some statistics with my expe¬ 
rience in 30 reviews over the last 10 years. In Table 1, first 
the type of review is shown using the classes ISO 14040, 
7.3.3; ISO 14040 7.3.2 and 'no fixed rules' (not exactly fol¬ 
lowing ISO, pre-ISO, company LCAs and similar studies). 
In the last case it should be remembered that the series of 
ISO standards was finished in the year 2000, but several 
reviews were performed in the years 1994-1999; more re- 


Table 1: Critical review (a personal statistic) 


Type of review 

Number 

[%] 

Total 

30 

100 


Method: 


Panel method acc. to 7.3.3 

19 

63 

Expert review acc. to 7.3.2 

6 

20 

No fixed rules 

5 

17 


Timing: 


A posteriori 

8 

27 

Accompanying/interactive 

16 

53 

In between/unclear 

6 

20 


cently, there have been no reviews with 'no fixed rules'. As 
can be seen, most reviews, where 1 have been involved, have 
been conducted according to ISO 14040, 7.3.3, followed by 
ISO 14040, 7.3.2 . 

Table 1 also shows that about half of the reviews have been 
performed in an interactive manner. 

The average number of experts per review is 2.8 for the 
panel method (7.3.3) and 1.1 for the external expert method 
(7.3.2). There are a few borderline cases between 7.3.3 and 
7.3.2, for instance if a large comparative study is not to be 
published. There is one published study from the US [22] 
which reports three reviewers and this is also the number 1 
got from an experienced reviewer in the States as the typical 
size of a critical review. There are a few LCA-studies with 
more than 4 experts, e.g. seven (+ 2 review-editors) in a 
large plastic waste recycling and recovery study performed 
in Germany [23]. 

The most interesting question regarding the actual costs of 
the review cannot be answered with certainty since the data 
are lacking. There is an old rule of thumb that the cost of 
the review should not be higher than about 10% of the total 
cost of the study. This may be a fair limit for large studies, 
but is certainly too low for medium size and small studies 
with a budget of, say, 50,000 € or less. This roughly corre¬ 
sponds to the results of a discussion at the Hannover Sym¬ 
posium 1999 [24]. 

5 Final Remarks 

According to ISO 14040, an LCA-study containing com¬ 
parative assertions that are disclosed to the public has to 
('shall') include a critical review according to the panel- 
method (§7.3.3). If the practitioner or the commissioner has 
additional comments (answers to the review comments), 
these, too, have to be published together with the main study, 
they are part of the whole report. This situation is clear, but 
a standard is not a law. An LCA-study can nevertheless be 
published if no claim is made that it has been conducted in 
(full) accordance with ISO 14040ff. The International Jour¬ 
nal of Life Cycle Assessment, for instance, will accept such 
manuscripts if they are reviewed by at least two referees and 
according to the generally accepted rules of peer reviewing 
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in scientific journals. This publication can, of course, be cited 
as a scientific paper, even as an LCA, - but not as an LCA 
generated according to ISO. This may be too rigorous in 
some cases, e.g. in scientific papers that use comparisons 
between product systems only for the sake of method devel¬ 
opment and evaluation. Are such results 'comparative as¬ 
sertions'? And do such LCA-studies need a critical review in 
order to be in accordance with ISO 14040ff? 

These questions may open a public debate. Letters to the 
editor and papers on actual critical reviews are welcome. 
Such a discussion may also give input to the ongoing revi¬ 
sion/restructuring of the ISO LCA-standards [25,26]. 
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